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BACKGROUND 

The  Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate  (AFRL/HEA)  has  been  conducting 
research  on  the  best  methods  to  use  in  automating  occupational-type  surveys.  One  area  of  particular  interest  has 
been  in  developing  valid  and  reliable  estimates  of  the  amount  of  time  that  individuals  spend  performing  tasks 
associated  with  their  jobs  (Albert,  Phalen,  Selander,  Dittmar,  Tucker,  &  Weissmuller,  1994;  Mitchell, 
Weissmuller,  Bennett,  Agee  &  Albert,  1995).  Automation  has  also  facilitated  exploration  of  new  methods  of 
gathering  estimates  of  actual  time  in  hours  and  minutes  spent  performing  each  task  by  each  respondent.  A 
number  of  field  applications  of  the  automated  survey  technology  have  been  accomplished  so  far.  A  key  finding 
across  all  studies  is  that  supervisors  and  subordinates  have  very  positive  reactions  to  the  software  as  a  data 
collection  tool.  Exit  surveys  show  that  instructions  are  relatively  clear,  automated  surveys  are  preferred  to 
paper-and-pencil  surveys,  time  to  complete  is  reasonable  and  typically  shorter  than  paper  and  pencil,  and  others 
in  the  career  field  can  be  expected  to  successfully  complete  the  survey  (Mitchell,  Weissmuller,  Bennett,  Agee  & 
Albert,  1995;  Albert,  Bennett,  Pemberton,  Holt  &  Wal droop,  1997). 

One  of  the  scales  that  has  exhibited  promising  results  in  gathering  this  information  is  the  Actual  Time  Spent 
(ATS)  Scale.  In  using  the  ATS  Scale,  each  respondent  is  asked  how  frequently  he  or  she  performs  each  task 
comprising  his  or  her  job  during  some  time  period  such  as  a  week,  month,  or  year  and  how  long  it  takes  on 
average  for  him/her  to  perform  the  task  once,  excluding  delays,  interruptions,  or  breaks.  The  product  of  these  two 
components  of  time  is  the  estimate  of  the  total  time  spent  by  the  respondent  on  that  task.  Then,  the  tasks  are  fed 
back  to  the  rater  in  high-to-low  order  on  total  time  spent  for  evaluation  and  revision  of  estimates  and  automatic 
recalculation  and  reordering  of  tasks  (until  the  respondent  is  satisfied  with  the  results).  Therefore,  the  actual  time 
spent  procedure  decomposes  total  time  into  its  two  basic  components:  "time  to  perform  a  task  once"  and 
"frequency  of  performance."  "Frequency"  has  been  shown  to  be  a  highly  reliable  counter  mechanism,  a 
perceptive  ability  that  is  consistently  more  accurate  than  that  which  governs  our  perception  of  time  (Hasher  & 
Zacks,  1984).  In  addition,  "time  to  perform  a  task  once"  has  the  advantage  of  representing  an  average  or  median 
value  rather  than  a  global  estimate  (Phalen,  1995).  For  example,  if  you  were  to  look  at  a  long  column  of  numbers, 
you  could  more  quickly  approximate  a  reasonably  accurate  mean  or  median  value  than  a  sum  for  that  set  of 
numbers.  The  ATS  scale  has  considerable  potential  for  assessing  the  impact  of  new  technologies  and  other 
operational  studies. 


OPERATIONAL  STUDY 

Based  on  their  extensive  experience  in  automating  both  occupational-type  surveys  and  general  purpose  surveys, 
AFRL  personnel  were  requested  to  evaluate  savings  or  costs  associated  with  the  introduction  of  portable 
computer  terminals  into  the  737th  Training  Group  (TRG)  at  Lackland  AFB,  and  to  determine  instructors’ 
attitudes  and  opinions  concerning  the  software,  hardware,  and  changes  in  their  work  processes  (Albert,  Bennett, 
Pemberton,  Holt  &  Waldroop,  1997).  The  737th  TRG  is  implementing  the  terminals  into  Basic  Military  Training 
(BMT)  to  automate  many  of  the  paper  and  pencil  processes  performed  by  their  instructors.  For  example,  each 
terminal  provides  access  through  a  network  server  to  the  instructor’s  lesson  plans,  relevant  Air  Force  regulations, 
standard  forms  that  the  instructor  needs  to  complete  on  a  regular  basis,  e-mail,  and  word  processing,  spreadsheet, 
presentation  graphics,  and  data-base  management  software.  To  make  the  transition  to  automation  smoother, 
AFRL  is  also  providing  expertise  in  optimizing  the  human-terminal  interface  and  efficiency  of  the  associated 
software,  and  in  designing  training  for  new  users  of  the  terminals.  In  addition  to  expanding  AFRL’s  experience 
with  computer-based  surveys,  this  effort  will  provide  a  good  test-bed  for  the  automated  ATS  scale. 

Basic  Military  Training  Instructors  (TIs)  have  employed  time-honored  and  well-established  training  procedures 
in  bringing  a  flight  through  Basic  Military  Training.  The  majority  of  TI  tasks,  however,  involved  paper  and 
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pencil  record-keeping.  The  737th  BMT  Squadron  at  Lackland  AFB,  TX,  had  an  opportunity  to  purchase 
hand-held  computers,  called  CruisePad,  in  an  attempt  to  automatic  many  of  these  processes.  The  CruisePad 
system  uses  FM  radio  transmitters  to  connect  the  CruisePads  with  a  host  computer.  Without  much  guidance, 
squadron  personnel  wrote  their  own  software  in  MS  Access  and  brought  a  few  of  the  CruisePads  on  line.  AFRL 
agreed  to  assist  with  identifying  and  solving  problems  associated  with  this  effort.  In  addition  to  providing  critical 
programming  assistance,  AFRL  completed  an  evaluation  of  the  impact  of  implementation  of  the  CruisePad 
technology  on  the  tasks  normally  performed  by  TIs.  The  primary  goal  of  this  Operational  Study  was  to  determine 
if  implementation  of  the  CruisePad  technology  has  an  impact  on  the  amount  of  time  TIs  spend  on  their  normal 
work  tasks.  It  was  hypothesized  that  there  would  be  an  overall  reduction  in  time  spent  on  tasks;  it  was  also 
anticipated  that  there  would  be  a  large  amount  of  variance  due  to  the  turmoil  created  with  the  implementation  of 
the  CruisePad  technology. 

Methodology 

With  several  experienced  TIs  serving  as  Subject  Matter  Experts  (SMEs)  and  using  the  Air  Force  Occupational 
Squadron-developed  TI  task  list  as  a  starting  point,  the  Institute  for  Job  and  Occupational  Analysis,  under 
contract  to  the  AFRL,  developed  a  126-item  task  list  which  was  incorporated  into  a  computer-based  survey  using 
the  Air  Force  Survey  Authoring  System  (AFSAS;  see  Mitchell,  Weissmuller,  Tucker,  Waldroop,  &  Bennett, 
1996).  An  "actual  time"  scale  (Albert,  Phalen,  Selander,  Dittmar,  Tucker,  &  Weissmuller,  1994;  Phalen,  1995) 
was  used  to  collect  time  estimates  by  incumbents.  IJOA  programmers  inserted  the  actual  time  software  into  an 
experimental  version  of  the  AFSAS  software.  In  addition,  several  open-ended  questions  were  included  with  the 
expectation  that  comments  from  CruisePad  users  would  assist  in  identifying  problem  areas  and  improving  the 
CruisePad  system. 

Identical  surveys  were  administer  on  two  occasions,  once  just  prior  to  the  full  implementation  of  the  CruisePad 
in  March  1998  and  three  and  a  half  months  later,  in  early  July,  1998.  During  the  intervening  period,  there  were 
several  significant  changes  to  the  software  (notably  the  inclusion  of  signature  recognition  software)  and  hardware 
(notably  the  FM  antenna  system).  Also  during  this  period,  a  number  of  TIs  utilized  both  the  existing  paper  and 
pencil  methodology  and  the  new  CruisePad  system  for  a  portion  of  that  four-month  period.  Additionally,  several 
technical  problems  identified  during  the  first  survey  were  in  the  process  of  being  addressed  at  the  time  of  the 
second  survey;  only  a  portion  of  those  problems  had  been  resolved.  Both  survey  administrations  were  proctored 
by  AFRL,  IJOA  and  BMT  staff  and  were  completed  in  several  group  sessions  held  in  a  computer  laboratory  in 
the  737th  BMT  Squadron  Headquarters.  The  first  sample  consisted  of  125  TIs  while  the  second  sample  consisted 
of  1 12  TIs.  Two  cases  were  eliminated  from  the  first  administration  and  one  case  was  removed  from  the  second 
study  due  to  response  patterns  that  suggested  that  the  respondents  misunderstood  the  requirements  of  the  survey. 
In  addition,  12  data  points  were  removed  from  the  first  survey  and  9  data  points  were  removed  from  the  second 
survey  after  being  identified  by  BMT  SME  personnel  as  providing  obviously  out-of-range  responses. 

RESULTS 

Data  were  summarized  using  the  Statistical  Package  for  Social  Sciences  (SPSS),  version  8.0  PC.  A  special  DOS 
utility  was  written  to  calculate  individual  time  estimate  responses  into  a  common  metric  -  hours  per  task  per 
year.  SPSS  was  utilized  instead  of  CODAP  for  technical  reasons.  The  background  and  exit  survey  data  were 
uploaded  using  the  AFSAS  software  while  the  actual  time  estimates  were  uploaded  using  a  special, 
IJOA-developed  version  of  the  actual  time  software  described  earlier.  Table  1  displays  the  results;  it  reflects  a 
decrease  in  time  spent  on  tasks  from  the  first  survey  to  the  second. 

Table  1  -  Comparison  of  Actual  Time  Estimates,  Survey  1  Versus  Survey  2 
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Mean  Hours  per 
Week 

SD 

N 

Mean  Hours  per 
Year 

SD 

71.38 

111.74 

125 

3711.70 

5810.53 

Survey  2 

68.35 

99.89 

112 

3554.18 

5194.18 

Difference 

3.03 

157.50 

A  similar  decrease  in  the  number  of  hours  incumbents  reported  working  per  week  (in  response  to  a  background 
item  asking  for  an  estimate  of  "hours  worked  per  week")  was  observed  (see  Table  2). 

Table  2  -  Comparison  of  Estimated  Hours  Worked  per  Week,  Survey  1  Versus  Survey  2 


Mean  Hours  per 
Week 

SD 

N 

Mean  Hours  per 
Year 

SD 

Survey  1 

74.98 

11.35 

125 

3899.17 

590.01 

70.79 

11.54 

112 

3681.32 

599.84 

4.19 

217.85 

The  ATS  data  document  average  timesavings  of  just  over  three  hours  per  week  per  TI  from  the  first  survey  to  the 
second,  or  about  157  hours  per  year  per  TI.  Table  2  reveals  that  when  estimating  the  hours  worked  per  week  (a 
more  global  estimate),  the  average  TI  said  he/she  was  working  about  4  hours  less  per  week.  Both  these  estimates 
are  in  the  same  direction  and  in  about  the  same  magnitude,  and  show  a  trend  towards  less  time  being  spent  on 
tasks  with  the  use  of  the  CruisePad.  Of  the  108  tasks  performed  by  the  majority  of  TIs,  56  showed  a  decrease  in 
actual  time  spent,  while  52  reflected  an  increase.  The  estimated  savings  may  be  an  underestimate  since  a  portion 
of  the  respondents  was  actually  using  both  paper  and  pencil  and  CruisePad  methodologies.  Table  3  shows  a 
direct  comparison  of  the  top  12  tasks  from  each  administration. 

Table  3  -  Top  12  Tasks  Compared,  Survey  1  Versus  Survey  2 
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Survey  1 

Task  Description 

Mean  Hours 
per  Year 

N 

N 

89 

Perform  Charge  of  Quarters 

277.23 

119 

24 

Conduct  inspections  of 
trainee  areas 

249.15 

(34.55) 

45 

Conduct  PC  training 

202.29 

107 

235.20 

87 

(32.91) 

34 

Conduct  or  participate  in 
flight  drills 

192.05 

100 

5.29 

83 

March  trainees  to/from 
training  activities 

175.67 

120 

33 

Instruct  trainees  on  1 

transitory  drills 

171.53 

H 

20.53 

51 

Conduct  evening  briefings 

146.84 

118 

25 

Perform  duty  dormitory 
inspections 

127.63 

123.39 

73 

4.23 

Document  trainee 
comments  on  Form  105(a) 

119.56 

121 

23 

Conduct  standby  dormitory 
inspections 

105.78 

(22.76) 

10 

Update  trainee 
administrative  records 

103.94 

121 

(2.70) 

8 

Administer  performance 
test  (not  physical  fitness) 

96.64 

16.18 

Considerable  information  was  also  provided  from  the  comment  section  of  the  survey.  Table  4  summarizes  these 
results.  The  detailed  Comments  file  has  been  forwarded  to  BMT  for  their  analysis  and  use. 

Table  4  -  Summary  of  Comments 


Major  Benefits 

Survey  1 

Survey  2 

Saves  Paper/Paperwork 

29 

26 

Easier  Access  to  Data  for  Viewing  and  Updating 

28 

20 

Saves  Time 

15 

18 

Easier  to  Track  Airman/data 

11 

12 

Keeps  Personnel  "Up"  with  Technology 

4 

4 

Email  Capabilities 

5 

6 

Software  (PowerPoint,  Word  etc.) 

5 

3 

Standardizes  Records 

2 

1 

Automatically  Updates  Records 

7 

1 

Record  Check  Option 

0 

10 

Centralized  and  Portable  Location  of  Data 

11 

7 

On  screen  options  (cut  and  paste,  spell  check) 

0 

17 

Easier  to  Post  Comments  and  Grades 

0 

18 

Total  117 

143 
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Major  Problems 

Survey  1 

Survey  2 

Reception/Antenna  Range 

54 

56 

Too  slow/time  consuming  (esp.  for  inspections  &  logon) 

43 

50 

Crashes/locks  up/kicks  user  off 

10 

30 

Loses/dumps  data 

6 

15 

Screen  size  too  small/glare/hard  to  read  in  bright  light 

14 

14 

Have  to  Print  each  comment  for  signatures 

12 

0 

Signature  Software  to  difficult  to  use/loses  signatures 

0 

35 

Record  check  is  difficult 

0 

20 

Too  bulky  to  carry/bad  carrying  case 

13 

4 

Hard  to  use/lack  of  training/not  user  friendly 

13 

7 

Can't  use  from  home 

10 

3 

Not  enough  printers/printer  locations 

5 

4 

Can't  view  whole  form  at  once 

4 

12 

Password  problems  for  Airman 

0 

6 

Keyboard  is  too  small 

0 

3 

Battery  life  too  short 

1 

2 

Two  people  can't  work  on  one  person's  data 

1 

1 

Total 

186 

262 

Suggested  improvements 

Survey  1 

Survey  2 

Improve  Reception/Antenna  Range 

32 

34 

Get  Signature  Software  (ex.  scan  signatures) 

17 

0 

Improve  Signature  Software 

0 

17 

Replace  with  Laptops/PC's 

7 

32 

Improve  Screen  -  size/visibility 

12 

10 

Improve  Processing  Speed/log-on  access  time 

14 

14 

Improve  Carrying  Case 

11 

2 

Provide  more  training/manuals/make  user  friendly 

12 

14 

Improve  Airman  Password  access 

0 

6 

Barcode  Use  for  inspections 

1 

0 

Swipe  Cards  for  inspections 

0 

1 

Internet  Access 

3 

2 

Battery  Life 

3 

1 

More  Printers 

6 

2 

Ability  to  save  or  back  data  up 

4 

6 

Put  more  forms  on  it/get  rid  of  all  paper 

7 

9 

Ability  to  view  a  form  w/out  having  to  scroll 

4 

15 

Larger  Keyboard 

2 

1 

Get  Rid  of  the  cruise  pad/go  back  to  paper 

4 

11 

Replace  antennas  with  satellite  reception 

1 

0 

Total 

140 

177 
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EXPERIMENTAL  STUDY 

The  primary  objective  of  the  experimental  phase  of  this  study  was  to  determine  if  the  respondents’  actual  time 
estimates  were  consistent  with  a  more  concrete  estimate  of  time  worked.  This  was  assessed  by  contrasting  their 
actual  time  estimates  with  their  responses  to  the  survey  item  capturing  their  global  estimate  of  "hours  worked  per 
week." 

Results 

A  comparison  of  actual  time  data  and  the  more  global  estimate  (survey  item  asking  for  "hours  per  week  worked") 
showed  both  estimates  to  be  similar  (see  Table  5). 

Table  5  -  Comparison  of  Actual  Time  Data  Versus  Global  Estimate  of  Time  Worked 


Hours  per  Year  from 
Actual  Time  Data 

SD 

Hours  per  Year  from 
Global  Estimate 

SD 

Survey  1 

3711.70 

5810.53 

3899.17 

590.01 

Survey  2 

3554.18 

5194.18 

3681.32 

599.84 

DISCUSSION 

Although  the  second  administration  of  the  survey  was  accomplished  before  many  important  "bugs"  had  been 
completely  worked  out  of  the  CruisePad  system,  the  operational  study  identified  a  trend  towards  a  decrease  in 
time  spent  on  tasks  with  the  CruisePad  technology,  as  hypothesized.  As  noted,  this  is  probably  an  underestimate 
of  the  total  timesavings  since  many  of  the  respondents  were  still  using  both  systems  at  the  time  of  the  second 
survey.  It  would  be  beneficial  to  re-administer  this  survey  when  the  CruisePad  system  has  stablized  and  TI's  have 
enough  experience  to  provide  realistic  estimates. 

The  experimental  study  confirms  that  ATS  estimations  were  consistent  with  a  more  global  estimate  of  time  spent 
working  on  tasks  (convergent  validity),  indicating  that  further  study  of  ATS  is  warranted.  As  evidenced  by  the 
high  standard  deviations  seen  in  both  surveys,  there  may  be  some  problem  with  how  the  ATS  data  collection  was 
operationalized.  Future  development  of  ATS  and  its  related  software  should  address  e  these  possible 
overestimation  issues. 

In  addition  to  the  current  operational  study,  BMT,  in  collaboration  with  the  Air  Force  Research  Laboratory 
(AFRL)  has  developed  a  prototype  electronic  classroom  where  the  impact  of  automation  on  student  feedback 
assessments  and  on  end-of-course  testing  is  being  assessed.  Customer  surveys  are  provided  to  students  at  the  2nd 
and  5th  weeks  of  training  to  track  satisfaction  and  to  identify  potential  conflict  areas.  Typically  done  as 
paper-and-pencil  surveys,  BMT  is  now  using  automated  data  collection  and  reporting  for  these  surveys.  For  the 
automated  testing,  AFRL  and  BMT  have  developed  a  preliminary  test  item  bank  and  an  automated  version  of  the 
current  paper  and  pencil  end-of-course  test.  Researchers  are  presently  applying  Item  Response  Theory  (IRT) 
methodology  to  develop  an  adaptive  testing  item  bank  for  future  implementation.  AFRL  has  designed  and 
implemented  a  comprehensive  evaluation  study  of  these  innovations  and  their  impact  on  BMT  work  processes, 
products  and  overall  efficiency. 
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